Virtualization technology is being widely adopted in grid and cloud computing platforms [31, 34, 23, 28] to improve server consolidation and reduce operating costs. On one hand, virtual machines (VMs) help improve security through greater isolation and more transparent malware analysis and intrusion detection [22, 24, 27, 10, 11, 14, 17, 29, 26, 19]. On the other hand, virtualization also gives rise to new challenges in maintaining security and privacy in virtualized environments. Although significant advances have been made in developing techniques to secure the execution of VMs, a number of challenges remain unaddressed.
VM checkpointing refers to the act of saving a permanent snapshot (or checkpoint) of a VM's state at an instant in time. Virtual Machine (VM) checkpointing enables a user to capture a snapshot of a running VM on persistent storage. A VM's state includes, at the minimum, its memory image and CPU execution state and possibly additional states such as virtual disk contents. The checkpoint can be later used for various purposes such as restoring the VM to a previous state, recovering a long-running process after a crash, distributing a VM image with a preset execution state among multiple users, archiving a VM's execution record, conducting forensic examination, etc. Most hypervisors such as VMware (VMware Inc.), Hyper-V (Microsoft, Inc.), VirtualBox (Oracle Inc.), KVM [35], and Xen (Xen.org) support VM checkpointing.
Despite the above benefits, VM checkpoints can drastically prolong the lifetime and vulnerability of sensitive information. Checkpoints are stored on persistent storage and contain the VM's physical memory contents at a given time instant. Data that should normally be discarded quickly after processing, such as passwords (especially clear text passwords), credit card numbers, health records, or trade secrets, can now be saved forever in persistent storage through VM checkpointing.
This vulnerability can be demonstrated using a common scenario of entering credit card information in a website. As shown in FIG. 4, the FireFox browser was started inside a VirtualBox VM. The browser was then connected to www.amazon.com, “my account” clicked to add credit card information, the number 9149239648 entered into the credit card number field, and then checkpointing was performed. When searching through the checkpoint file with a hex editor, the credit card number entered earlier was located. In some of experiments, the checkpoint file contains the string “addCreditCard-Number=9149239648”, which can enable an attacker to locate the credit card number easily by searching for the string “CreditCard” in the checkpoint. Furthermore, even if the checkpointing is performed after the browser terminates, the credit card number can still be located in the checkpoint file, likely because the browser's memory was not cleared before the browser terminated. In other words, the common advice to “close your browser after logging out” may give users a false sense of security. Many users are not aware that their input data may still reside in memory even after the application that has processed such data terminates. Such users may mistakenly assume that checkpointing the VM is safe simply because the application has terminated.
Passwords in the memory of xterm terminal emulator (both running and terminated) were also identified in the checkpoint file.
Besides memory, even checkpointing a VM's disk may also end up storing users' confidential data in the snapshot. For example, Balduzzi et. al [36] analyzed 5303 public Amazon EC2 snapshots and found that many of them contain sensitive information such as passwords, browser history, and deleted files.
Previous work on minimizing data lifetime has focused on clearing the deallocated memory (also known as memory scrubbing). Chow et al. [6] and Garfinkel et al. [12] discussed in depth the problem of sensitive data being stored in memory, and observed that the sensitive data may linger in memory for extended periods and hence may be exposed to compromise. In [7], authors proposed a multi-level approach to clearing deallocated memory at the application, compiler, library, and system levels. A similar mechanism is included in Windows operating systems, which uses system idle time to clear deallocated memory pages [30]. Also, in Unix systems, it is common to clear memory before reuse [12]. However, simply clearing deallocated memory does not solve our problem because memory pages that have not been deallocated may contain sensitive information and such information may be checkpointed. As a result, SPARC also clears the memory pages of the excluded processes in checkpoints. Selectively clearing memory pages during checkpointing is much more challenging than scrubbing only deallocated memory because multiple processes may share the same memory pages (e.g. shared libraries) and we must ensure that excluding one process will not affect other processes when the VM is restored.
Garfinkel et al. [12] also proposes to encrypt sensitive information in the memory and clear the sensitive information by simply discarding the key. However, encrypting sensitive information in memory can add significant overheads to access the information and may still expose sensitive information if the VM is checkpointed at the moment when some program decrypts the sensitive information.
Features protecting virtual disk, memory, and checkpoints have found their way into research prototypes as well as commercial virtualization products. Garfinkel et al. [13] developed a hypervisor-based trusted computing platform that uses trusted hardware features such as encrypted disks and the use of a secure counter to protect against file system rollback attacks, to permit systems with varying security requirements to execute side-by-side on the same hardware platform. The platform's privacy features include encrypted disks and the use of a secure counter to protect against file system rollback attacks in which the state of a file is rolled back. [15] and [2] also suggested encrypting checkpoints. However, encrypting the checkpoint alone is insufficient because (1) it still prolongs the lifetime of confidential data that should normally be quickly destroyed after use; (2) when the VM is restored, the checkpoint will be decrypted and loaded into the memory of the VM, thus exposing the confidential data again; (3) the checkpoint file may be shared by multiple users, thus increasing the likelihood of data leakage.
VMware ACE [2], VMware Infrastructure [33], and VirtualBox [25] allow users to exclude the entire memory from being checkpointed. However, none of them provides a level of granularity that we do by selectively excluding processes from the checkpointed memory. Davidoff et al. [9] retrieved clear text passwords from the physical memory of a Linux system. Their work aimed to show that the physical RAM may retain sensitive information even after the system has been powered off, and the attacker with physical access to the system can steal information through cold boot memory dumping attacks. However, with checkpoints, the problem is significantly more severe: in the RAM, the amount of time the sensitive information persists in the memory after the machine is powered off, is limited by the RAM's ability to retain information in absence of power. However, the checkpoints are saved to the disk and the information stored in the checkpoints can persist for long time. Also, they assume that the attacker has physical access to the system, but we do not.
Several prior works have employed VM checkpointing to enable execution replay for intrusion analysis and OS debugging. Dunlap et al. [11] proposed an intrusion detection mechanism called ReVirt which allows instruction-by-instruction replay of the guest OS execution. King et al. [18] used ReVirt and disk logging to implement an OS debugger. However, neither work attempts to address data lifetime issues raised by VM checkpointing.